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ABSTRACT 

This pamphlet contains five reports on different 
aspects of the Oalcleaf Small Computer Project, The major objective of 
the project was to individualize elementary mathematics instruction 
by using a fully-integrated system of computer-assisted instruction, 
evaluation, and class management. The curriculum used was based on 
IPI Mathematics. The reports given here provide detailed accounts of 
the operation of the system; student access to the machine, 
generation and scoring of tests, record-keeping functions, and luany 
other topics are discussed • A study of the effects of practice at a 
computer terminal on the acquisition and retention of computational 
algorithms is also reported • (SD) 
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Abstract 



The Oakleaf Small Computer Project is designed to show that, in 
schools where individualized instruction is already taking place, a small 
computer system can help those schools become more adaptive and effec- 
tive. V/ithin this project, work has proceeded on developing a suitable 
computer time-sharing system, computer-aided instruction and testing, 
and a computer data management system. One desirable outcome of this 
project would be a coherent instructional system that would bring all those 
components together. This set of papers describes mathematics compo- 
nents of the project and shows how this integrated system might evolve. 
The papers were presented as a symposium at the American Psychological 
Association's 81st Annual Convention in Montreal, Canada, 1973. 
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INTRODUCTION 



Robert Glaser 

When a new technology appears on the scene with an apparent poten- 
tial for improving education, the new technology can be handled in two ways. 
One way is to immediately list the technology's virtues, state how it can 
improve things, mount som:* quick displays of its use, turn it into some- 
thing marketable, and sell it to the schools where it will be used to im- 
prove the quality of education. The technological object itself becomes 
the focus of concern. How can it be used is the paramount question. This 
way of dealing with a new technology seems to have been the mode employed 
in recent years with educational television, free-standing teaching machines, 
programmed instruction materials, and computers in the schools. When 
introduced in this way, the technology generally becomes the tail that wags 
the dog; cultists arise who advocate use of the new technology, and the new 
technology becomes a discipline of its own, relatively uninfluenced by rele- 
vant or needed knowledge. The technological entity becomes the central 
thing, rather than becoming an object to be integrated into a knowledge 
system where its use can be systematically exploited and studied. When 
introduced in tliis way, there is a great crescendo of e::citement, writing, 
funding plans, followed by a gradual petering out of enthusiasm, a mounting 
disappointment, and wonderment about what went wrong. Final questions 
to be asked include: Is the new technology really usefv^l? Are we r.;ady 
for it yet? 

A second way in which new technological developments can be 
handled is to work with them in the context of disciplines that can nurture 
them and to which they mighf-, in turn, contribute. This generally means 




that aspirations for immediate breakthroughs take a back seat while the 
power, utility, and force for change of the new object are allowed to 
emerge through study of failures, little successes, and the demonstra- 
tion of utility in relatively small-scale intensive application. The accu- 
mulation of evidence of potential stimulates some avenues of use and 
discourages others. What happens in this latter mode of technological 
development is that scientists, engineers, curriculum developers, and 
practitioners begin to bring their knowledge, their problems, and their 
questions in contact with the potential of the new technology. Experimen- 
tal give-and-take occurs, and the results obtained are informally or for- 
mally assessed as is appropriate to the situation. 

In the course of this experimentation, different parties benefit 
a*^ different times. Sometimes, education in the schools seems to be 
nf^glected because some computer system problem seems important to 
solve, or some theory of learning or instructional technique needs to be 
assessed, or some way of reorganizing the classroom and the traditional 
school day needs to be simulated to see how feasible this would be if a 
computer were introduced. At the same time that these things go on, 
pilot attempts in cooperation with practical school people and curriculum 
developers are being conducted. 

This symposium reports on somj^ possible uses of a computer 
resource in the elementary school. Hie way this work has proceeded 
can be generally characterized by the second mode I have just described. 
The project has two fundamental goals: 

(1 ) To examine the ways in which computer technology could be 
used to assist Ln making teaching and the environment for learning more 
personalized and individualized and less subject to regimentation and lock 
stepping . The terms "personalized" and "individualized" these days have 
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the sound of empty catchwords, but in a deep sense, they represent the 
long-term aspirations of educators to teach according to needs of indi- 
vidual students, a goal which has been continuously and widely espoused 
but extraordinarily difficult to achieve in the context of our conventional 
school structures. The achievement of individualized environraonts for 
learning is a serious matter requiring more detailed Ueve'opmental work 
and less general nonoperational talk* 

(2) To design a computer system (software and hardware configura - 
tions) that can facilitate and encourage the innovative use of computer tech - 
nology in elementary schools (and in schools in general )* Toward this end, 
the project has designed and is investigating the capability of a medium- 
oized computer that is inexpensive and manageable, but still efficient and 
large enough to serve as a local facility for a school or for several schools 
in a system on a shared basis. A local computer (with perhaps some infre- 
quent network assistance) appears to have reasonable possibilities for rela- 
tively immediate adoption by schools in the near future. The costs of small 
computers are decreasing and eventually might be within the range of a 
capital investment for a small school system. A local computer eliminates 
the expense and awkwardness of long-line communication, eliminates the 
necessity for an investment in a computer personnel organization, and 
incorporates the virtues of a system dedicated to a special purpose. With 
the development of sophisticated and ingenious software, a relatively small 
computer might well serve the needs of a school for computer technology* 
In the long run, when computer utilities are as common as electric power 
utilities., plugging into a large system might be the most efficient way to 
go, but at the moment, it appears that innovation will be encouraged by 
attention to small system design and investigating its capabilities for serv- 
ice to a school. 
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To accomplish these two purposes, several lines of investigation 
were simultaneously started* The intention was to investigate different 
. aspects of computer use in some detail, and to concentrate these efforts 
in a way so that they all would converge upon a particular group of students 
and teachers. This means keeping within the limits of available resources 
by working on a number of computer services in a small and intensive w«iy, 
rather than carrying out the extensive development of one overall curricu- 
lum or one kind of service that would cover a whole school. In general, 
the tactic was to investigate in a few classrooms a raage of different possi- 
ble uses of computer technology. As these various uses are assessed, an 
overall system for services and research can be pUnned. 
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A COMPUTER -ASSISTED INSTRUCTIONAL SYSTEM 



Jerome Rosner, Tse-Chi Hsu, and Gilbert Puente 

This paper has two purposes: (1) to describe the school environ- 
ment in which the Qakleaf Small Computer Project operates and how tiie 
school-project interface was effected; and (2) to describe a student per- 
formance data management system that is intended to enhance the adaptive 
qualities of an educational environment. 

Oakleaf is a small (300 children, kindergarten through grade six) 
elementary public school located in the Baldwin-Whitehall diotrict of 
suburban Pittsburgh. It has served as an LRDC developmental school 
since 1964. As such, most of the instructional programs used in the 
school are individualized. The intermediate mathematics program cur- 
rently in use--IPI Mathematics--has been described in detail elsewhere 
(see Lindvall Bolvin, 1967). The newly designed primary curriculum-- 
Individualized Mathematics (1971)--is an updated adaptation of the lower 
levels of IPI math. Both programs are baped upon the following general 
instructional model (from Glaser, 1970): 

1. "The goals of learning are specified in terms of observable 
student behavior and the conditions under which this behavior is to be 
manifested.'* In other words, the student's instructional goals are defined 
as behavioral objectives with accompanying, clearly stated criteria. 

2. "When the learner begins a particular course of instruction, 
his initial capabilities --those relevant to the forthcoming instruction — 
are assessed. Simply stated, pretests are used to make certain that 
the student is asked to learn (a) something he does not already know, and 



(b) something he should be able to learn with reasonable effort, given his 
present knowledge. 

3. "Educational alternatives suited to the student's initial capa- 
bilities are presented to him. The student selects or is assigned one of 
these alternatives. " This acknowledges the importance of flexibility and, 
to some extent, learner autonomy- -that more than one route is available 
toward achieving competency in a general domain of learning. 

4. "The student's performance is monitored and continuously 
assessed as he learns. " That is, tests are provided to measure the 
accomplishment of small increment instructional goals as well as for 
major milestones. 

5. "Instruction proceeds as a function of the relationship between 
measures of student performance, available instructional alternative::, 
and criteria of competence. " Simply, progress is noted as the student 
achieves instructional goals and demonstrates these competencies by 
meetinfy the precisely stated criteria of the curriculum tests. 

6. "As instruction proceeds, data are generated for monitoring . 
and improving the instructional system. " This, of course, acknowledges 
that an instructional system must always be studied and modified to meet 
existing conditions --that no instructional system continues to be satisfac- 
tory if it is static and nonresponsive to the dynamics of the environment 
in which it is used. 

Translated into operational terms, the mathematics curriculum 
used at Oakleaf comprises a hierarchy of instructional goals defined as 
behavioral objectives. The objectives are organized into units, each unit 
representing a specific subdomain of skills --such as addition, subtraction, 
multiplication, and division --at a specific level of difficulty. Criterion- 
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referenced tests have been constructed for each objective, thus providing 
both teacher and student with a clear statement of ihe desired behavior 
and the criteria tor demonstrating mastery. Both the teacher and student 
are kept aware of the student's progress in a way that can be reliably 
recorded and monitored by using an organized strategy of: (1) placement 
testing- (to. determine the Svudent's appropriate level and unit--'*where he 
is"); (2) pretesting (to determine what the student already knows within a 
particular unit); (3) providing relatively self-instructional teaching materi 
als to teach the student what he has not yet learned; (4) testing of short- 
term goals (individual skills) with curriculum-embedded tests (CETs); 
and (5) postte sting of units. The flow diagram shown in Figure 1 illus- 
trates the te sting -teaching procedures of individualized instruction at 
Oakleaf School. 

The computer serves three functions at Oakleaf: testing (CAT-- 
computer-assisted testing), instruction (CAI- -computer-assisted instruc- 
tion), and management (CMI-- computer -managed instruction). The three 
are obviously interrelated; each is capable of providing meaningful input 
for others and, in turn, each is responsive to the input directed to it in a 
way that foretells the eventual possibility of the system operating indepen- 
dent of external supervision. For example, as CAT procedures are able 
to provide reliable data regarding a student's competencies and needs, it 
is feasible that CAI will provide the necessary experiences to teach those 
skills, and that CMI will serve as a monitor of the outcomes of both func- 
tions, transmitting information in both directions as well as to those out- 
ride the system who will use the data to refine the workings of the system 
itself. Only aspects of the CMI component will be discussed in this paper 
the other two will be addressed by other papers in this symposium. 
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Placement test taken 



Select one unit for study 
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Take the pretest 
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YES 
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NO 





Select one skill for study 








1 




¥ 


Additional diagnosis 
of the student's 
weakness for the 
same skill 




Prescribe appropriate 
instruction among 
alternatives based on 
the student's needs 



Study according to 
the presc'intion 



Take a CET (Curriculum 
Embedded Test) for this skill 



Additional diagnosis of 
the student's weaknesses 
for the same unit 



NO 



YES 




4 




Take a post test 
for this unit 






Correct any errors 
made on the pretest 



Figure 1. Instructional and testing procedures at the Oakleaf School. 
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Computer -Managed Instruction 

A major goal of CMI is to provide teachers with up-to-date, on- 
line information for makmg instructional decisions. To accomplish this, 
a common file was designed to accumulate all pertinent daily performance 
data for each student, such as the skills he has worked on and is currently 
working on, the nature of the instruction (CAI, paper /pencil)^ the tests he 
has taken, and test outcomes* 

Given this data base, the management system provides for three 
types of communications between the users and the system (Hsu, Fox, & 
JLerner, 1973). Tlie first pertains to adding new records to the data base, 
modifying records, or deleting , records. Three computer programs 
("GATHER," "UST," and "UPDATE") are designed for this purpose. * 
At this stage in the development of the system, all data are collected 
interactively from students, teachers, or classroom aides. As men- 
tioned above, when the system has been thoroughly tested and debugged, 
modifications will be made to the CAI and CAT programs currently on- 
line so as to trani5m'-t the necessary data directly and automatically at 
the end of an activity. 

A second type of communication is the routine printing of reports 
that have been shown to be of general interest. Several such reports have 
been selected for initial development. Ihese include: 



a. REPORl--this report consists of a list of students who 
have mastered at least one skill in a specified time 
period (from one to ten days) and a list of students 
who have not mastered at least one skill. 

b. REPOR2--this provides a list of students who have 
failed the same CET or posttest more than once for 
current units. 
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c. STREPl and STREP2-. these reports tabulate the fre- 
quency of stude.nts in each unit of the two math curricula, 
Individualized Math (IM) and Individually Prescribed 
Instruction (IPI) math, 

d. GCSUM--this is a summary of units comploted for the 
period specified' and is useful for preparing report 
cards. 

e. lDDATA--the purpose of this program is to permit a 
listing of the roster for a specific ID. The roster con- 
sists of latest achievement test scores, IQ scores, 
and units completed to date for each student. 

f. DAYSUM--this program calculates the mean and 
standard deviation of the numbor of days spent in 
each skill of a particular unit. The information is 
reported for each grade and is a grand total of all 
grades. 

Tlie third t-ype of communication is Inquiry Response ("QUERY"). 
This deals with procedures for making requests from the system for a 
specific piece or block of information that is not available through the 
prepared report generating programs described above. For example, an 
inquiry might be made to the system asking the names of students who 
took the computer pretest for a particular multiplication unit on Septem- 
ber 20. An example of QUERY output is shown in Figure 2. 

These report programs (and others still in the planning stage) all 
share one major feature: They provide updated information reg?.rding 
each student's performance, thus enabling the teacher (and other interested 
individuals) to identify quickly (1) those students who are "bogged down" 
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DO YOU WANT INSTRUCTIONS? *•* 



A. WHEN "HEADING" APPEARS FROM TABLE 1, TYPE ATTRIBUTES 
(SEPARATED BY COMMAS) THAT YOU WANT PRINTED. TYPE THEM 
IN THE ORDER YOU WANT THEM PRINTED ACROSS THE PAGE. DO 
NOT SPACE. 

B. WHEN 'TITLE" APPEARS, TYPE A REPORT TITLE OR JUST HIT 
CARRIAGE RETURN. 

C. .WHEN "1." APPEARS, TYPE AN ATTRIBUTE, A COMMA, AND ONE OR 
MORE VALUES (SEPARATED BY COMMAS). LIMIT: 10 VALUES. 

D. THE COMPUTER WILL CONTINUE TO PRESENT YOU WITH 2,3 

FOR EACH OF THESE ENTRIES, FOLLOW THE PROCEDURE IN C ABOVE. 
THESE ARE THE DETERMINANTS OF YOUR RETRIEVAL. WHEN YOU 
HAVE MADE ALL SPECIFICATIONS YOU DESIRE, TYPE "GO" IN 
RESPONSE TO ONE OF THE NUMBERS PRESENTED. 



HEADING: IOANE, ©^JAPC, ClA5S, DATE, SKiLU 

TITLE: Puf lis WHo HAVC FAILCP ComiTff^ CCT W 5>C»tL % Cf £ Pt/ipE 

1. UiotT, EDlv^ 

2. SKltL, ? 

3. ACTlV, CCET 

4. DEClS, /JMftS 

5. G-O 



NAME 


GRADE 


CLASS 


DATE 


SKILL 




^^TEST 


DECIS 


8ELL, SALLY 


5 


2 


48 


8 


7 


10 


NMAS 


BELL, SALLY 


5 


2 


5t) 


8 


4 


10 


NMAS 


BELL, SALLY 


5 


2 


52 


8 


7 


10 


NMAS 


BROOKS, TOM 


4 


1 


53 


8 


6 


10 


NMAS 


COX, JIM 


5 


2 


62 


8 


7 • 


10 


NMAS 


GREEN.TIM 


4 


1 


73 


8 


7 


10 


NMAS 


SMITH, RUTH 


4 


2 


94 


8 


7 


10 


NMAS 


SMITH, RUTH 


4 


2 


97 


8 


6 


10 


NMAS 


WOODS, JOHN 


5 


2 


99 


8 


7 


10 


NMAS 


WOODS, JOHN 


5 


2 


100 


8 


6 


9 


NMAS 



Figure 2. An example oi QUERY output. 



11 



somewhere; (2) a weak component in the curriculum where students* 
progress tends to slow down, causing a "pile-up"; (3) those students 
whose classroom performance over time is sufficiently erratic to war- 
rant close investigation; and (4) those students who are sufficiently pre- 
cocious in their performance to suggest the initiation of a supplementary 
enrichment program. Hence, although its major function is to facUitate 
implementation of an individualized instruction model, CMI makes it pos- 
sible to effectively monitor the progress of the entire group and to search 
for performance contrasts within and across individual students over time. 
The need for this in a responsive environment is evident; withouf such 
management resources the instructional system becomes less responsive 
and less individualized. 

Yet another facet of CMI is providing suggested alternative pre- 
scriptions to the teacher based on the student's test performance-- 
especially the nature of his errors. Still in the development stage, the 
ultimate goal of this facet is to identify effective instructional materials 
or activities for every math skiU included in the curriculum. Standard 
teaching sequences and computer-assisted instruction programs are 
obvious alternatives. In addition, where the need is indicated, teachers 
will prepare their own instructional materials. Eventually, all of this 
information will be stored in the computer where it will be available to 
present suggested prescriptions to a student inunediately following testing. 

Conclusion 

During the first two years that the computer was used at Oakleaf, 
the terminals were housed in a small conference room, isolated from the 
rest of the school and supervised by a paraprofessional aide who per- 
formtd all procedures that preceded and followed a CAI or CAT session, 
that is, LOGIN, calling in the desired program, LOGOF, etc. The 
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students merely answered questions, in a sense using the cathode-ray 
tube (CRT) as an answer sheet in a workbook. The teachers die not "use" 
the facilities, they merely allowed their students to go to the terminals 
when they were summoned by the developers. As a result, the teachers 
gained little benefit from the computer and, indeed, viewed the whole 
enterprise with some distrust and perhaps even hostility. 

The situation was altered last year. The terminals (10 CRTs, 4 
TTYs [teletypes]) were relocated in learning centers--central spaces 
that are adjacent to and surrounded by classrooms --thus, making the 
computer programs much more public and available. More important, 
a procedure was designed and installed that enabled the teacher to decide 
which of the students would go to the terminals for testing and/or instruc- 
tion. 

Under this new system, the teacher provides the student with a 
prescription (a flow diagram specific to the task he is to engage in at the 
terminal). Figure 3 illustrates a typical prescription. It tells the stu- 
dent what program output will appear on the CRT, what signal will cue 
his response, what response is required from him to access the desired 
program, and so forth. Aided by this flow diagram, the student is able 
to initiate the desired program, participate in its activities (being aware 
of what he is to do in response to program requests), terminate it at an 
appropriate time, and return to his classroom with informaHon regarding; 
his performance recorded on the flow diagram sheet. In every grade but 
the first (and here, too, many are able to perform capably), the Scudent:. 
were able to perform independently after a brief indoctrination, except 
in instances where unforeseen technical difficulties were encountered. 
The computer, in effect, has become an integral component of the school 
environment- -an instructional resource that can be exploited by the teacher 




FACTS 




JfEAsl'SeiN: (student number)®! 



r RUN®! 



TYPE THE NAME OF THE PROGRAM YOU WANT TO RUN 



EFACTS®r 



TYPE THE NUMBER OF THE OPERATION YOU WANT 

1 ADDITION 

2 SUBTRACTION 

3 MULTIPLICATION 

4 DIVISION 



E 



ft Tnis will occur 
^/ after the first 



(10 minutes of work) 



THAT'S ALL FOR NOW 

GOODBYE 

PLEASE LOG OFF 



STOP WD 
/E: EXIT 



1. 



J SWITCH \ 



Figure 3. Example of a CM) prescription. 
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as (s)he guides the daily learning experiences of each student, and by the 
students as they gain awareness that they are the controllers, rather than 
the servants, of technology. 

The ultimate worth of computer applications in an elementary 
school has yet to be determined, but our experience to date indicates that 
it has the potential to be an extremely useful component in the construc- 
tion of an adaptive educational environment- -an environment that is opti- 
mally responsive to all of its students. 
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COMPUTER -ASSISTED TESTING- 



Tse-Chi Hsu, Marthena Carlson, and Kurt Pingel 



ie: 
time 



In an individualized instruction system, criterion-referenced tests 
are used extensively to diagnose the characteristics and to determine the 
mastery status of the learner. Hiere are some potential weaknesses of 
these tssts administered through the traditional paper and pencil mode 
(1) The number of items and their level of difficulty are fixed at the tir 
of construction; (2) criteria for mastery or nonmastery is fixed; (3) be- 
cause there are tests for every objective, a child's time involved in test- 
ing is considerable; (4) tests may be too short to be -reliable if only limited 
numbers of items are used for the purpose of reducing testing time; (5) 
management of testing and scoring is laborious; (6) feedback of test results 
to the child is not immediate; and (7) using a previously taken test is occa- 
sionally required, thus raising questions about practice effect. 

In a computer-assisted instructional system, it is logical to use 
the computer for testing as well as for instruction. Rather than simply 
replacing existing paper and pencil tests with computer-administered tests, 
it is desirable to improve measurement procedures and overcome some of 
the deficiencies of paper and pencil testing. Some of the ways a computer 
can be used to improve measurement procedures are: (1) test more skills 
in a shorter time; (2) perform a more detailed diagnosis without being 
limited by fixed difficulty levels and numb.;r of items; (3) increase the 
validity of the decisions of the students' mastery status with reference to 
a specific skill; (4) provide precise diagno£.tic information and immediate 
feedback; and (5) generate additional nominally equivalent tests for each 
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skill. In other words, the purpose of using the computer for testing is to 
promote adaptive testing by taking into account individual differences. 
Reaching the desired level of efficiency necessary for adaptive testing is 
easier in the computer mode than it is in the paper and pencil mode. The 
advantages and potential of adaptive testing are discussed extensively by 
Weiss and Betz (l973). 

This paper presents cur experiences in using computer-assisted 
testing within an individualized instruction system. Discussion is focused 
on the design of the computer-assisted testing model and psychometric 
issues associated with the implementation of the model. 

The Computer-Assisted Testing (CAT) Model 

The computer-assisted testing (CAT) model (Ferguson & Hsu, 
1971) is designed for a unit of instructional content that consists of 
several behavioral objectives (skills). The model is intended to perform 
pretesting, postte sting, curriculum-embedded testing (CETJ, and to 
generate exercise pages (practice problems) for all of the skills included 
in the unit. Basically, the model consists of four components: (1) item 
generation based on the requirements of the skills; (2) branching strate- 
gies based on the nature of the content; (3) item administration and scoring 
according to required formats, and (4) decision-making about the mastery 
status of the student. A multiplication unit, shown in Figure 4, will be 
used to illustrate the structure of the model and the development of com- 
puter testing programs based on this model. 

Item Generatio n. This component generates the numbers specified 
by the item forms developed for an objective. An item form is a general 
specification of a behavioral domain. The actual test items which repre- 
sent that behavioral domain are categorized by a list of generation rules. 
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These rules are used to generate nominally parallel items (Hively, Pat- 
terson, ^< Page, 1968). Normally, three or four item forms are developed 
for a behavioral objective. However, the number of item forms developed 
for an objective depends on the donruxin of the objective. 

'fhe item forms for the CAT model were established by content 
analysis and by examination of the existing test materials to determine 
the types of errors students tend to make within each objective. Properly 
constructed item forms should be helpful in prescribing appropriate instruc- 
tion. For illustration purposes, the description of three possible item 
forms for Objective 3 in Figure 4 are listed next. 

Objective 3: Given a multiplication example in which both 
factors are two-digit numerals, the student 
multiplies using the standard algorithm. 

Form I: Vertical. All digits in the 
multiplier and multiplicand 
will be 1-5. 

Form 11: Vertical. The digits in the 

multiplicand will be 1-5 when 
the digits in the multiplier 
are 6-9 or vice versa. 

Form III: Vertical. All digits in the 
multiplicand and multiplier 
will be 6-9. 

The unique requirements for each form are specified in the pro- 
gram. When a test for this objective is requested, nunibers will be 
generated randomly to meet the specifications of the item form sequen- 
tially from Form I to Form HI, one at a time. This procedure? assures 
the efficient generation of a computer test because actual items do not 
need to be stored. The use of adequate item forms assures sampling of 
the domain, and random generation of the digits within forms guarantees 
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nominally parallel tests, not identical item for item, but similar because 
the replacement digits have a controlled range. 

Branching Strategy , This component controls the sequence of 
testing. In order to provide branching that takes into account the per- 
formance of the student, branching sequences must be established for 
each unit and programmed into the model to manage the flow of testing. 
The following considerations were used to determine the branching 
sequences. 

(a) A valid hierarchy was assumed. Therefore, when a 
child passes a higher objective, the lower objectives 
within the same branch are assumed mastered. For 
example, in Figure 4, Skills 2 and 3 are assumed 
mastered if Skill 4 is passed. 

(b) All terminal behaviors in the unit are tested before 
certification of mastery is made. In Figure 4, 
Skills 1, 4, 5, and 7 are terminal objectives. 

(c) In a larger unit, say more than 10 skills, testing 
should begin near the middle of the hierarchy so 
that branching can be either way, and testing time 
can be reduced. 

Item Administration and Scoring . After the required numbers are 
generated in component 1, this component presents the item according to 
prespecified formats and computes the key for the item. When the student 
enters his response, it is matched with the key. The student is given a 
message if the response is correct. If the response does not match the 
key, the key is printed. The number of correct responses is recorded 
internally for decision-making purposes. 

Decision Making . The last component of this model is the decision- 
making strategy. In CAT, the strategy for determining the mastery or non- 
mastery of a skill is based on Wald's Sequential Probability Ratio Test 
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(Wald, 1947), A detailed discussion of the application in computer testing 
is providea by Ferguson (1971). The following is a brief summary of the 
concept. 

This test is computed from four variables: 

(a) Pq -- an arbitrary criterion for mastery; 

(b) Pj an arbitrary criterion for nonmastery; 

(c) a -- Type I error (tolerance for misclassifying 

a mastery student as a nonmastery student); 

(d) $ Type II error (tolerance for misclassifying 

a nonmastery student as a mastery student). 

Consider the determination of the mastery status for an individual 
as the testing of the following hypotheses: 

The questions governing the sequential process in Bernoulli Trials are: 
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" l"^ - ^ then accept H^. 



" V^i] - ^ then accept H^. 
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(c) If Tie < I 1—1 1 — then take an 

additional observation* 

where: s is the number of successes, 
f is the number of failures. 
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Taking logarithms, the last expression becomes: 
The equations giving the boundaries of this keep- sampling region are: 

A graphical representation of these boundaries is given in Figure 5. For 
example, using the decision boundaries for the revised CAT program (solid 
lines), a student who answers seven of eight items correctly and then 
answers the ninth one con-ectly, is classified as a mastery student. If the 
student did not answer the ninth item correctly, he is administered another 
item. The lower limits of the decision boundary illustrate that no student 
may master a skill with less than five items tested or achieve nonmastery 
with less than two items tested. . 

Actual programming of this model into a computer testing program 
is illustrated in Figure 6. This flowchart shows the relationship between 
these four components. After the student turns on the terminal and 
requests a program for a specific unit, the program a^^ks for the student's 
name and the type of test- -either a pretest, GET, posttest, or exercise 
page. For CETs and exercise pages, the student is asked to enter the 
number of the objective he wants. For pretesting, the student begins with 
the skill prespecified in the branching chart as shown in Figure 4. If this 
test is a continuation of a previous test, the student begins with the skill 
interrupted previously. 
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Start 



Request student's name and 
his Intent (CET. PRE, POST, 
or EXERCISE PAGE) 



Request starting objective 



YES 



NO 



(Generate an item for \ 
this ohjective / 



Want an EXERCISE 
PAGE for another 
objectiva ? 



^^y^Pfint the ^tefI^O^^^ 

and key. Are \j Intent » EXERCISE 



NO 
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PAGES ? 



) 



NO 



\ Presents and scores the item N 

> r / 



(Decides \ 
proficiency status / 
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^ Intent » CET ? 



Pass or Fail 
NO 



NO 





YES 
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Record results. 




Print CET 


summary 









(Is this the last \ 
objective in the unit V 





YES 

r 




Record results. Print 
Pre/Post summary 







/ Was there an Y^ll ii 
interrupt ? messag 



Note. 



mterrupt 
message 



NO 



Print "GOODBYE" 
message 



►^Stop^ 



> means information inside 
_ ish 



s handled by a subroutine. 



Figure 6. The computer-assisted testing model (adapted from Hsu et al.. 1973). 
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When a test for a specific objective is requested, the program 
randomly selects numbers from the range specified by the item form* 
Items are constructed from these numbers according to the required for- 
mat and presented to the student* The computer waits for a response and 
then checks the accuracy of the response* Jf the response does not match 
the key, the key will be printed before the next item is generated* In the 
case of exercise pages, the program continues to generate items and keys 
for later scoring until it reaches the numbers specified in Table 1* 



TABLE 1 

Prcspecifled Number of Items for an Exercise Page 



Number of Item Forms Number of Items Total Number of 

inaSkill For Each Form Items 

lor2 5 5orl0 

3 4 12 

4 3 12 

5 or more 2 10 or more 



After each item is scored, the number of correct responses is 
cumulated Into the record for decision-making using Wald's Sequential 
Probability Ratio Test* If continued testing is indicated, the program 
will branch back to item generation for the same objective using the next 
item form in order. If the decision can be made, either pass or fail, the 
prog am will print the summary statistics or proceed to the next skill in 
the cas'2 of pre- or posttesting. The summary statistics classify the stu- 
dent's responses according to item forms. This summary is useful to 
teachers for prescribing appropriate instructional material* 
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Psychometric Issues 

Based on the CAT model described above, experimental programs 
were written for E level units of IPI Math in addition/subtraction, multi- 
plication, and division. (The classification of IPI Math units is given in 
the first paper of this symposium. E level units are comparable to ele- 
mentary arithmetic for third or fourth grade. ) Field testing of these pro- 
grams at Oakleaf School began in Fall 1971. Major activities in the 197U 
72 school year were to collect data for improving CAT (Carlson & Hsu, 
1973), and to explore the procedures for evaluating the quality of the tests 
generated and administered by a digital computer (Hsu & Sebatane, 1973). 
The evaluation effort for the 1972-73 school year focused on the compari- 
son of the CAT group and the P&PT (paper and pencil testing) group in 
terms of average time spent in taking each test, average number of days 
spent in the unit, scores on a criterion test, and students' attitudes toward 
math, testing, and the computer. Overall evaluation effort is still under- 
way so that no conclusive results can be presented here. Only some psy- 
chometric issues associated with CAT will be discussed below. 

Mastery Criteria. The sequential decision strategy for mastery 
status is an important characteristic of CAT. This' strategy allows the 
number of test items to vary according to the child's performance. If a 
child is consistently wrong or right, an earlier decision of either mastery 
or nonmastery can be made than is presently possible with paper and pen- 
cil tests. If the pattern of responses is not a sequence of all right or all 
wrong, more items are administered. The additional items should permit 
a more valid decision than would be possible with paper and pencil tests. 

In 1971-72 the values used in the decision strategy were arbitrary 
mastery, p^ = .85, arbitrary nonmastery, p^ = .60, Type I error, a = . 20, 
and Type 11 error, 6^.10. If a decision was not made in 30 items, it was 
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forced after considering the proportion correct for those 30 items. In 
comparing the number of mastery decisions made under the CAT criteria 
with those from the P&PT criterion (85 percent mastery), the conclusion 
was that students were less likely to achieve mastery under the CAT cri- 
teria. In addition, the teachers felt that a maximum of 30 items per skill 
was too high. Although there were only 10 of 104 tests where more than 
20 items were administered, we decided to modify the decision rules so 
they would be more comparable with the paper and pencil decision, and 
hopefully, make the teachers feel more comfortable using CAT, Thus, 
for 1972-73, the arbitrary nonmastery criterion (p^) was reduced to .55, 
This change narrowed the region of continued testing, as shown in Figure 
5, and faster decisions of either mastery or nonmastery could be made. 
In addition, the maximum number of items to be tested for each skill was 
reduced to the same number of items presented on exercise pages shown 
in Table 1, 

The arbitrary values used in our decision strategy are not final in 
any sense. They should be further evaluated in terms of the nature of con 
tent, the ease of transfer to the next higher order skills, and students* 
previous background and interest levels, A Bayesian sequential approach 
is currently under investigation. This approach can take into account the 
difficulty of the skill and the student's previous history (Hsu & Ping el, 
1974), 

Quality of Computer Testing , The problem in computer testing is 
not that programming for testing is a difficult task, but that the procedure 
for evaluating the quality of the test have not been developed. The unique 
characteristics of the tests are that two students taking the same type of 
test for the same skill do not get the same items. The decision about the 
mastery status of the skil? needs to be made without reference to other 
students; traditional test evaluation procedures are not designed for this 
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type of test. Therefore, one of our efforts is to explore the procedures 
for evaluating the quality of tests which are generated and administcr^^d 
by the computer. 

An empirical study was conducted to compare P&PTs administered 
to a group of students v/ith equivalent computer-generated tests adminis- 
tered to another group of students. Hiis is essentially a study of matched- 
item testing, where every student takes the same items (P&PT); and un- 
matched-item testing, where every student takes different items (CAT) on 
the variables of item form difficulty, item form discrimination, and gen- 
eralizability coefficients (Hsu & Sebatane, 1973). Item form difficulty is 
defined as the proportion of correct responses for items generated by 
that item form. Item form discrimination is defined as the difference 
between the proportion of correct responses in the mastery group and the 
proportion of the correct responses in the nonmastery group for all items 
generated by that item form. Stratified alpha coefficients were computed 
for paper and pencil tests (matched-item). Stratified gamma coefficients 
were calculated for computer-generated tests (unmatched-item). 

The results show that the item forms for computer-generated tests 
are consistently more difficult than that of equivalent paper and pencil 
tests. However, the way itom forms discriminate between mastery and 
nonmastery groups are about the same for both types of tests. In terms 
of generalizability, the stratified gamma coefficients of computer-generated 
tests are slightly higher than the stratified alpha coefficients of equivalent 
paper and pencil tests. An important conclusion of the study is that the 
item form as a unit for evaluating the quality of an unmatched-item test 
seems reasonable. In short, there are many issues along this line in 
need of further exploration. For example, the index of item form homo- 
geneity is another possible procedure for evaluat».ng the quality of computer- 
generated tests (Macready & Merwin, 1973). 
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Te sting Hierarchy , The efficiency of CAT is dependent on an effi- 
cient branching procedure, determining the branching sequence, a 
valid hierarchy for the unit is assun>ed. If this assumption is violated, 
an invalid decision about mastery status may occur. Thus, the evalua- 
tion of testmg hierarchies is another important psychometric issue. 

In order to determine if the hierarchy used in CAT was valid, the 
paper and pencil pretest scores were collected. Because paper and pen- 
cil pretests test all skills, a score is available for each objective and, 
therefore, provides evidence to investigate the assumptions made in es- 
tablishing the branching procedure used in computer testing. Followin,^ 
a technique of modified scalogram analysis as discussed in Cox and 
Graham (1966), the scores on each objective were coded pass if the per- 
cent correct was 80 percent or above, and fail if less than 80 percent. If 
the hierarchy is valid, students should not master skills that are terminal 
skills unless they also master the subordinate skills. 

For the branching pattern used in the multiplication unit shown in 
Figure 4, the questions are: Do students who pass Skill 4 also pass Skills 
3 and Z, and do students who pass Skill 7 also pass Skill 6? Only three of 
41 (7 percent) who passed Skill 4 did not pass one of the lower skills and 
two of 41 (5 percent) who passed Skill 7 did not pass Skill 6. Thus, we 
feel the testing hierarchy for this unit is reasonably valid. 

Time Factor . Our interest in time involves two related issues. 
The first issue is time spent working on the test. The second issue is 
the amount of time it takes students to work through a unit. To study 
these issues, students were randomly assigned to one of the two test 
situations: CAT or P&PT. 

In terms of time spent taking the test, our aim is to eliminate 
unnecessary testing if the decision can be made quickly; otherwise, more 
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Items should be administered. In other words, making an accurate deci- 
sion is more important than the reduction of time during testing. Never- 
theless, we wish to keep the testing time within a reasonable range. This 
year the attempt to collect data for the paper and pencil group was unsuc- 
cessful and we have only the time required for the computer test group as 
shown in Tables 2 and 3. However, the average number of items required 
for a pre^ or posttest is substantially smaller than that of P&PTs, which 
have 42 items. We suspect that the time spent taking computer tests is 
less than that spent taking paper and pencil tests. 



TABLE 2 

Average Number of Items and Average Time Spent in 
Minutes in Taking a Pre- or Posttest by the CAT Group 



Posttest 



Unit 

E Addition/Subtraction 
E Multiplication 
E Division 



Pretest 

Number of Items Time 

21.2 47.4 
26.1 42.4 
29.6 42.7 



Number of Items Time 

22.2 49.3 

21.0 32.7 

29.0 41.2 



Table 3 shows the average number of items and average time spent 
for typical CETs. The computer CETs, on the average, test more items 
than the six-item P&PTs. The question, "Is there any advantage in testing 
more items on computer CETs?" was raised. This can be answered by the 
data obtained from posttests. This data showed that 81 percent of the stu- 
dents who passed a computer GET passed the first posttest on that skill. 
Only 44 percent of the students who passed a paper and pencil GET were 
able to pass the skill on the first posttest. 
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TABLE 3 

Average Number of Items and Average Time Spent in Minutes 
in Taking a GET for Skills in E Multiplication Unit b/ the CAT Group 



Skill 


Average Number of Items 


Average Time 


1 


36/5-7.2 


47/5 ■ 9.4 


2 


17/2 - 8.5 


19/2-9.5 


3 


36/4 " 9 


64/4" 16 


4 


43/5 « 8.6 


55/5" 11 


5 


83/11-7.5 


163/11 - 14.81 


6 


15/3-5 


17/3-5.6 


7 


64/7-9.1 


140/7-20 



To determine -f there is a differenco between the two groups in 
the amount of time it takes students to work through the unit E Multiplica- 
tion, the number of days to mastery was collected for each student. Using 
the number of skills a student was required to work as the covaiiate, an 
analysis of covariance was performed. The computer F-statistic is sig- 
nificant at 10 percent level (F - 2. 90, df = 1, 42, p = . 0961 ). We suspect 
that there are three possible interpretations. The first is the better 
validity of the decision criteria for mastery status used in computer tests. 
Tlie better validity means that CAT students are more likely to pass the 
posltest and will not be required to restudy any portion of the unit. The 
student who fails a pobttest will be required to restudy and, therefore, 
\v 111 take more time to master the unit. Another interpretation is the time 
spent taking the test. For those students with a clear-cut mastery or non- 
mastery » the decision can be made very quickly. These students will not 
spend unnecessary time testing and can proceed with Instruction in an 
appropriate skill without any delay. A third interpretation is that a more 
specific diagnosis results in a more accurate prescription, thus reducing 
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time. Tliese intcrpTctations arc all tentative, further information about 
the time opent in instruction and accurate nica.»>urcb of time spent taking 
paper and pencil tests bhould be obtained to support these mterpretations. 



Summary 

This paper describes the structure of the computer testing model 
and discusses the psychometric issues associated witli the implementation 
of such a model in a system of individualized instruction. Because ihc 
evaluation effort is still underway, no conclusive generali/.ation can yet 
be made. However, there are several remarks that can be made here. 

First, our effort to make CAT one of the alternatives for testing 
under the present IPI system soems to be working. Tlie students from 
third grade and above hive few problems handling the terminal. TJie com- 
puter can generate desired tests and make valid decisions about mastery 
status comparable with the paper and pencil tests. Some teachers arc 
uncomfortable having students leave their class to use the computer be- 
cause of the loss of personal contact. However, the majority nf Mie 
teachers are willing to prescribe CAT as a testing alternative. Tliey 
find the test summary and exercise pages generated by the computer 
useful for prescription. 

Second, our preliminary results show that students on the aver- 
age can maoter a unit faster when CAT is used. Tlus is probably due to 
the reduction of testing time and valid decisions about masteiy status. 
More inipurtant, this tentative result implies that computer testing can 
be more adaptive to ip'Uvidual differences. 

Finally, the use of the computer in testing has a potential for 
modifying the traditiuiial concept of measurement. Althoitgh the coni- 
puter alone may not administer an idoal tc^t at this stage, audio and 
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graphical presentation of items should be possible in the near future. A 
combination of these applications can niake traditional paper anci pan^.il 
tcfif* obsolete. On the other hand, traditional measurement concepts are 
focused on matched item testing. With the development of indir-idualized 
instrucf-^n and criterion-referenced measurement, the need for stuslents 
to be tesf by the same item^ no longer exists. The administering of 
unmatchc J'item testing by a digital computer should have great appeal, 
ir this statement is accurate, we can anticipate the growfh of unmatched- 
item testing and the development of measurement theories associated with 
this type of testing. 
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LEARNING OF COMPUTATIONAL ALGORITHMS 
THROUGH COMPUTER PRACTICE 



Eric Jacobson 

My particular concern is how students learn computational algo- 
rithms and how we can get students to learn these algorithms. Computa- 
tional algorithms consist of a series of separate operations which, when 
considered in detail, can be very extensive. It can be presumed that their 
acquisition is not an instantaneous process but proceeds gradually across 
time as the student learns more and more detail of each operation. -This 
internal acquiaifcion is reflected at least crudely in the probability that a 
test problem will be computed correctly. 

Of special interest for education is the terminal stage of this 
acquisition process. It is a goal of math instruction to produce students 
who henceforward will always, or nearly always, compute right answers. 
Thus» if this goal is to be met, the acquisition process must continue not 
simply to a point where the student is fairly accurate but to where the 
student is 100 percent accurate and to where it is reasonably certain that 
he v/ill retain this capajbility for 100 percent accuracy. 

Information required to learn an algorithm can be presented to a 
student in one of three modes. First, it can be given as a verbal descrip- 
tion of the algorithm, so, for example, the student can read or be told 
how to multiply. Second, lite student can watch a demonstration of the 
algorithm; that is, he can watch a teacher multiply correctly. Third, 
he can perform a potentially correct version of the algorithm and receive 
feedback concerning its correctness. Or, in other words, he caii prac- 
tice multiplying, 
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A priori reasoning strongly supports the notion that verbal and 
demonstration modes should be used relatively early in acquisition and 
practice relatUely late. A complete novice would have little likelihood 
of generating a correct algorithm and thereby learning from practice. 
Only after verbal instructions and demonstrations have imparted some 
minimum amount of information does it seem likely that practice could 
be effective. Furthermore, according to long standing educational belief, 
the practice mode is important during the final phase of » cquisition when 
the student achieves 100 percent accuracy and high retentiveness. 

The above reasoning was used to develop and apply computer- 
presented practice at Oakleaf. Practice was given late in the acquisition 
period and it was reasoned that this practice would be especially useful 
in helping students achieve perfect accuracy and in aiding long-term 
memory of the algorithm. The late-acquisition period was easy to define 
in the individualized Oakleaf system as the period after the passing of the 
unit posttest. At this point each student has answered at least 85 percent 
of the problems correctly for each skill. 

Retention as a goal presents an especially difficult problem, 
since the prime measure of acquisition available is present-time accuracy. 
If a oladent is working at 100 percent accuracy, yet still has not practiced 
enough to retain the algorithm, how can it be determined when enough prac- 
tice has occurred? One solution to this retention problem is to give all 
students large amounts of postmastery practice. This insures t.hat all 
students will get enough practice to retain the algorithm. This strategy 
has the obvious shortcon.lng that some students will be practicing much 
more tlian they need to. It might be possible to effect a savings in the 
effort pv*t into practice by postponing practice after initial acquisition and 
applying it as remediation only for students who have failed to retain this 
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algorithm. In this way students who reach a high retentive state without 
practice need not waste time in practice. This strategy of delayed prac- 
tice would be efficient only if it can serve to reinstate lost information 
relatively quickly and easily. If students cannot relearn easily from prac- 
tice alone, then the delayed practice would not be efficient. In order to 
answer some of these questions some students were given immediate long« 
term practice while others werd given smaller amounts of practice delayed 
several weeks atter initial acquisition. , 

Method 

. - , instruction, Computer practice programs^ were written to-prev - 
sent problems from two multiplication units in the IPl Mathematics curricu- 
lum (see Rosner et al. , in this symposium). Unit D introduced the algo- 
rithm for multiplying multi-digit factors and led ultimately to the multi- 
plying of a two-digit number times a two-digit number. Unit E continued 
work on the multiplication of larger integers, and introduced the commuta- 
tive and associative laws for multiplication. Most students working in D 
were fourth graders; most working in E were fifth graders. 

Each unit is divided into skills which must be mastered indepen- 
dently by the student. There are 13 skills in the D level unit and seven 
skills in E level. No practice was provided for three skills from unit D 
and two skills from unit E. These skills dealt with terminology, exponen- 
tial notation, and the representation of arrays of objects as multiplication 
equations. Of those skills which were practiced, two skills in D and one 
in E could be identified as number facts only (single-digit products), six 



Computer programs used were designed and written by Kenneth 
J. Hernandez. 
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skills in D and two in E as multi-digit algorithm only and two skills in D 
and two in E neither. Included were such topics as the special properties 
of the factors 1 and 0 and the associative principle. 

The practice program was based on hierarchies of problem forms 
taken from the units. Student success on one form caused the program to 
present problems of the next most difficult form up to the most difficult 
level in the unit, while failure caused the program to move back. This 
decision procedure meant that a successful student .»rould remain at the 
top of the hierarchy practicing the most difficult problems from the unit. 
The algorithm used to solve these difficult problems subsumed as compo- 
tientb' the ulgorithms used -to -solve th« easier probleme. - 

Within a problem form, individual problems were produced by 
get e rating numbers randomly to nieet the specifications of the form. The 
student copied the problem from the cathode- ray tube terminal onto paper 
with pencil, computed the answer on paper and then typed his answer back 
in at the terminal. The program then gave him feedback based on speed 
and accuracy of solutiou. If the student answer was wrong, the correct 
answer was shown and the student was asked to copy it. After the feed- 
back interaction was complete, a new problem was shown. Practice ses- 
sions lasted ten minutes. 

P rocedure . In order to master a unit in the curriculum the stu- 
dent must show independent niaster> of all the skills in the unit. Tlius, at 
the point of niaijtery when a student entered the experiment, he had shown 
at least 85 percent proficiency for each skill in the unit. The maintenance 
groxip received three practice sessions a week for six weeks. The remedial 
group received equivalent nonpractice, nonmultiplication computer experi- 
ence for three ten-min4ite sessions a weok for four weeks and then three 
practice sessions a week for two weeks. Tlie no-proctice group received 
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the equivalent nonpractice, nonmultiplication experience for three ten- 
minute sessions a week for all six weeks. Twelve weeks after unit mas- 
tery and six weeks after completing computer work, all students were 
given the unit posttest again as a test of retention. Tlie structure of the 
experiment is shown graphically in Figure 7. 

While in the computer phase of the experiment, students left their 
classroom during the math period and went to the terminal where an aide 
helped them get onto the program to which they were assigned experimen- 
tally (either practice or nonpractice). The standard schedule was compu- 
ter on Monday, Wednesday, and Friday, but if for any reason a student 
could not attend a regularly scheduled session, it could be made up on a 
Tuesday or Thursday. In this way all students received 18 sessions over 
the six- week period. Retention tests were given in. the classroom by the 
teacher. 

Subjects. In the individualisied Oakleaf setting, students completed 
the target units and became eligible for the experiment separately through- 
out the school year. Each new set of three students entering the experi- 
ment was randomly assigned to the maintenance, remedial, and no-prac- 
tice group. This random assignment was performed separately for D 
level and E level students. Because the final triad was not completed and 
because one retention test could not be administered, the number of sub- 
jects in each group was not equal. Tlie number of subjects in each group 
for each unit level is shown in Table 4. 

TABLE 4 

Number of Students in Each Group and Level 





Maintenance 


Remedial 


No Practice 


0 


10 


9 


10 


E 


5 


4 


5 


38 



^ ■ 45 



TREATMENT 
GROUPS 



MAINTENANCE 



NO-PRACTICE 



REMEDIAL 



Practice 



Nonpr?cticp 
k ^ 



Nonpractice Practice 




4 



• 4 Weeks - 



6 Weeks - 



UNIT POSTT EST 
MASTERY 



■6 Weeks - 



TREATMENT 
cNDS 



RETENTION 
TEST 



Figure 7. Graphic representation of experimental treatments. 
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Results 

A ccuracy in Practice . In order to remove the effects of varia- 
bility due to program familiarization, the data from the first practice 
session were removed from the analysis. The remaining data were 
grouped into three session blocks (two sessions in the last block) yield- 
ing six blocks for the maintenance {^roup and two blocks for the remedial 
group. Each block is the rough equivalent of a week's work. An error 
in the data retrieval system caused the loss of some data from terminal 
sessions, so some of the data blocks represent incomplete results. 
Table 5 contains the average percent correct and average answer latency 
for each session block for D and E level maintenance groups separately 
together. Hie first two lines in this table show the number of student 
sessions for which data were available in each block. Table 6 contains 
comparable data for the remedial groups. 

TABLE 5 

Average Accuracy and Latency Across Session Blocks 
for Maintenance Group 



Session Block 


1 


It 


III 


IV 


V 


VI 


Student Sessions 
in Block 
D 


30 


30 


30 


28 


24 


10 


E 


15 


15 


15 


15 


14 


5 


Percent Correct 
D 


63 


69 


71 


67 


68 


60 


E 


63 


70 


74 


64 


79 


64* 


Total 


63 


69 


72 


67 


71 


62 


Mean Latency 
:n Seconds 
0 


57 


47 


46 


44 


40 


52 


E 


75 


60 ' 


49 


47 


41 


44 


Total 


61 


50 


47 


49 


40 


50 




TABLE 6 



Average Accuracy and Latency Across Session Blocks 
foi Remedial Group 



Session Block I 



Student Sessions in Block 

D 27 15 

E 12 7 

Percent Correct 

D 56 69 

E 74 75 

Total ' * 61 ' '70 

Meao Latency in Seconds 

D 56 51 

E 54 40 

Total 55 46 



In the maintenance group, practice seemed to have very little 
effect on accuracy across blocks, with no block deviating greatly from 
70 percent correct. Separate repeated measures analyses of variance 
were performed on D level subjects and all subjects combined, using 
phase as the treatment variable and using an arcsin transformation of 
the percentage scores. Only the first four phases were used since 
only these phabes had data for all subjects. Both analyses yielded 
F- values of less than one (df = 3, 1 0, F = . 78; df = 3, 1 5, F = . Z(>) indi- 
cating no significant differences between phases in accuracy. 
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In the remedial group the four-week interval apparently pro- 
duced a decrement In accuracy, at least in D level where the accuracy 
is 56 percent in block 1. By block II, however, this deficit has disap- 
peared and the percentage correct is Indistinguishable from the main- 
tenance group. Repeated measures analyses of variance for D level 
subjects alone and all subjects together indicated that this difference 
between phases in the remediation group was not significant (df = 1,8, 
F = 3. 95; df - 1, 12, F = 1.43; both p > , 05). 

The mean percentage correct for both maintenance and remedial 
groups is consistently below the 85 percent level at which each student 
had mastered the curriculum unit. In order to assess the significance 
of this difference, the 95 percent confidence intervals about the prac- 
tice means were computed. These intervals were established for D 
level alone and D and E level together, for maintenance blocks I through 
IV separately and these blocks taken together, and for remedial blocks 
I and II separately and taken together. Computations were done on the 
arcsin transformations of the percentages and the resulting interval 
limits were then translated back into percentages. These intervj.ls 
are shown In Table 7. The curriculum criterion of 85 percent falls 
within all intervals. Since a student can repeat posttests until he 
reaches the criterion level, it might be supposed that the 85 percent 
measure would be a biased estimate of the students *s real knowledge. 
It seems likely that the students* real ability was close to the 70 per- 
cent level demonstrated in practice and that given that level of ability 
a student would have a good chance of putting together a run of correct 
answers sufficient to pass the posttest. 



42 



TABLE 7 



95 Percent Confidence Intervals About Average Accuracy 
Means for Session Blocks and Totals 
(in percent) 





Lower 
Limit 


Mean 


Upper 
Limit 


□ Students Only 
f^AINTENANCE 
Session Block 








1 


36 


67 


92 


II 


25 


71 


99 


III 

IV 


32 
43 


71 
70 


97 
90 


Total 


35 


70 


95 


■KEfv^SDIAL 
Session Block 
1 

ti 


• 

26 
47 


69 


oD 
87 


Total 


34 


63 


88 


D and E Students 
maintpnanpp 

Session Block 










32 


66 


93 


II 


32 


71 


97 


III 


40 


72 


94 


IV 


43 


69 


90 


Total 


34 


69 


95 


REMEDIAL 
Session Block 








1 


27 


64 


93 


II 


45 


71 


91 


Total 


35 


67 


93 



Latency in Practice . Answer latency declines across practice 
blocks for both maintenance and remedial group& (Tables 5 and 6). A 
repeated measures analy&is of variance was performed on the mair- 
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tenance scores for D level students alone and for D and E level sttklents 
together using block as the treatment variable. Only the first four blocks 
for which there was a full complement of student data were included. 
Both analyses (D alone, and D and E together) indicated that the decline 
in latency was significant = 3, 10, F=8.83, p < . 0 1 ; df = 3, 1 5, 
F = 45.84. p < .01). A similar pair of analyses for the two remedial 
blocks yielded a significant F- value for D and E students together 
(df = I, 12, F = 8. 79, p < . 025), but an F- value which was not signifi- 
cant for D level students alone (df = 1,8, F = 4. 532). 

Retention . Retention scores can be examined in terms of the 
number of correct answers or in terms of the number of separate skills 
within the test for which the 85 percent mastery criterion Wws met. 
Both breakdowns are given in Ttible 8 for D level tests and. Table 9 for 
E level. These tables also show the results for certain .subsections of 
the entire test, including the objectives practiced in the program, those 
objectives in the unit but not practiced, objectives involving only the 
multi-digit algorithm and those involving only number facts. Entries 
in these tables are the number of answers correct, or skills passed, 
over total number of problems, or skills, and the resultant percentages, 
for all students within each group. 

The story told in these tables is consistent and clear: There is 
no difference between any of the groups in any of the observed components 
of retention. There are some very small percentage differences in favor 
of the hypothesis that practice increases retention, e.g. , 6 percent for D 
level skills passed and 8 percent for E level correct answers. Even 
these small differences, however, cannot be considered evid.,nce in sup- 
port of practice, since they are just as great for nonpracticed objectives 
as for practiced ones. 
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TABLE 8 

Correct Answers and Objectives Passed for D Level Retention Tests 



Maintenance 



Remedial 



No Practice 



Answers 
Correct 



Total 
Practice 
NoPractice 
Algorithm 
Number Facts 



1037/1210 

86% 
839/960 

87% 

198/250 
79% 

445/550 
81% 

233/240 
97% 



035/lCSO' 

86% 
758/864 

88% 

177/225 
79% 

398/505 
79% 

210/216 
97% 



1027/1210. 
85% 

823/960 
86% 

204/250 
82% 

432/550 
78% 

228/240 
95% 



Objectives 
Passed 



Total 
Practice 
No Practice 
Algorithm 
Number Facts 



102/130 
78% 

75/100 
75% 

27/30 
90% 

39/60 
65% 

19/20 
95% 



89/117 
76% 

69/90 
77% 

20/27 
74% 

35/54 
65% 

17/18 
94% 



93/130 
72% 

68/100 
68% 

25/30 
83% 

35/60 
58% 

18/20 
90% 
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TABLED 

Correct Answers and Objectives Passed for E Level Retention Tests 



Maintenance 



Remedial 



No-Practice 



Answers 
Correct 



Total 
Practice 
. No Pr«:tice 
Algorithm 
Number Facts 



351/425 
83% 

239/265 
90% 

112/160 
70% 

56/60 
93% 

149/150 
99% 



282/340 
83% 

192/212 
90% 

90/128 
70% 

42/48 
87% 

120/120 
100% 



318/425 
75% 

237/265 
89% 

81/160 
51% 

53/60 
88% 

149/150 
99% 



Objectives 
Passed 



Total 
Practice 
No- Practice 
Algorithm 
Number Facts 



13/35 
51% 

17/25 
68% 

!/10 
10% 

9/10 
90% 

5/5 
100% 



16/28 
57% 

12/20 
60% 

4/8 
50% 

6/8 
75% 

4/4 
100% 



18/35 
51% 

17/25 
68% 

1/10 
10% 

9/10 
90% 

5/5 
100% 
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Discussion 

In general, practice had little effect on student performance. 
Practice failed to cause an increase in accuracy of algorithmic compu« 
tation and it produced little or no increase in the retention of the algo- 
rithm. TJus was so for the small doses of practice given the remedial 
group and the larger doses given the maintenance group. Practice did 
cause increase in the speed of calculation, but compared with accuracy 
and retention, this effect must be considered of lesser educational impor- 
tance. 

Tlicro are certain jdifflcultij^s interp:tetipg the fetejation tesl^ 
results. Oiie indication of this difficulty is the fact that students scored 
higher on the practiced objectives in the retention test than they did during 
practice. This. anomaly can be explained by the fact that the hierarchy^ 
based practice j>rogram tended to give students the most difficult problems 
whereas the retention test sampled all levels of difficulty uniformly. 
Nevertheless, despite this explanation, it is not possible to determine 
exactly how much forgetting, if any, took place. 

The possibility that, in this study, a ceiling effect obscures the 
influence of practice might be entertained. The no-practice group did 
eo well on the retention test that it might be argued tliat the practice 
groups could not show any superiority. We rejcci tha: argument by 
pointing out that in terms of the ultimate educational objective, these 
students were not close to the coiUng. About 70 percent of the practiced 
objectives were passed at the 85 percent criterion on the retention test. 
While this does indicate considerable learning and retention, it is defi- 
nitely below the school expectation of 100 percent. Tlius, although it 
cannot be determined whether the deficit was in initial learning or in a 
failure to retain what was learned, it is clear that these students had not 
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fully acquired an ability to use the multiplication algorithm accurately, 
and that practice did nothing to aid this acquisition. 

Negative results such as these do not, of course, allow unequivo- 
cal interpretation. Perhaps practice does not have the beneficial effects 
initially hypothesized. Or perhaps the particular situational parameters 
used in this practice were wrong and the right kind of practice would have 
helped. Admitting that the decision procedures of standard methodology 
do not allow us to decide between these two interpretations, yet still 
requiring a decision in order to proceed, we chose the former interpre- 
tation as most reasonable. That is, we decided that our practice was as 
good as any and reformulated our ideas about the role of practice in gen- 
eral. In defense of this interpretation it can be said that no independent 
measure of program quality, in the domain of stvdent attention and moti- 
vation and teacher evaluation indicated that there was anything wrong with 



Here then is our tentative model of the learning of computational 
algorithms. This model is consistent with our results, and other data, 
and is guiding our next attempt at algorithm training. According to our 
model there are two phases in acquisition. In the first phase a student 
acquires an algorithm to a high proficiency, being able to compute accu- 
rate solutions for something like the 85 percent rate required in IPI. In 
phase two the student continues to increase his accuracy rate to an asymp- 
tote near 100 percent and succeeds in storing the routine in memory such 
that it is permanently retained. 

At present we know very little about this second phase. Educa- 
tional folklore has it that practice will bring about phase two acquisition 
but our data indicate that it will not. Other studies which have shown 
increases in the accuracy of long computational algorithms due to prac- 
tice (e.g. , Suppes ^ Morning star, 1972) do not coi.cradict our findings 
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or interpretation, ^incc the final student accuracy rate in these studies 
does not approach 100 percent. 

One explanation of phase two is that it is not acquisition at all, 
but a general lowering of error rate for all cognitive processes. During 
phase one a istudent learns an algorithm perfectly but fails to respond at 
100 percent accuracy because of random errors occurring in the execu- 
tion of the algorithm. Later, and by a process entirely independent of 
the knowledge of the algorithm, this general error rate is reduced (per- 
haps by the acquisition of self -checking routines, perhaps simply through 
maturation of the nervous system). The result would be what we have 
been calling a phase two increase in solution accuracy. 

This explanation is not relevant to putative changes in long-term 
retention occurring during phase two. Perhaps this type of retention 
occurs only through very long-term spaced practice extending over 
several years. Another possibility is that It is a result of a reorganiza- 
tion of the storage structure of the algorithmic knowledge making it 
permanently retrievable. Speculation on phase two acquisition is ham- 
pered greatly by the lack of normative data on accuracy and retention in 
adults. It may be the case that adults do not perform above a mean of 
60 percent correct and except for intermittent review through application 
would not retain an algorithm indefinitely. 

Students in our study were practicing during phase two, that is, 
after they had already reached the 85 percent accuracy level. This 
practice haci little or no effect on accuracy or retention. The s^wdents 
who practiced in studies reported by Sappes and hii co-workers (Suppes 
& Morningstar, 1972) were practicing daring phase one. They worked in 
. nontndivtciuahzed svbtcmb m v^hich individual mastery wa^ not and could 
not be assured. Therefore, when they began practicing they were still 
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in phase one, and practice aided acquisition. Examination of pre - and 
posttests scores reported by Suppes and Morningstar (1972) bear this 
interpretation out. Although there was demonstrated improvement for 
each of the 114 practice block-school units used in their comparison 
(Chapters, Table 1, pp. 208-210), for only 51 such units was the final 
accuracy 80 percent or better, and for only 14 was the accuracy 90 per- 
cent or greater. 

Thus, we cannot expect large amounts of practice administered 
after initial acquisition to either produce a further increase toward com- 
plete accuracy nor to increase retention. Practice, however, occurring 
during initial acquisition can aid that acquisition. Suppes and Morning- 
star have shown that practice is a sufficient condition for learning but 
not that it is either a nocessary or optimal condition. We return to an 
earlier assertion that early in learning verbal explanations and demon- 
strations are relatively more efficient means of transmitting informa- 
tion. It may turn out that practice is not necessary at all for acquisi- 
tion, or at least that it has an optimal effect in very small amounts. 

Future Work . At the present we are working on what we have 
called phase one acquisition, attempting to take beginning students and 
bring them up to a reasonably high level of responding. At the outset 
we make two pr<*suxnptions* First, optimal learning will require a 
sequencis and/or mix of different instructional modes and, second, there 
will be individual differences between students in the optimum mix or 
sequence of modes. 

There are actually four instructi,mal modes we can make availa- 
ble to students. First, we can prepare short booklets that describe 
verbally the operation of the algorithm. Second, we have written com- 
puter routines that perform the algorithm in real time on a cathode-ray 
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tube and thereby demonstrate to the student the steps o£ the algorithm. 
Third, we have our standard practice programs that call for a student 
to use an algorithm and receive feedback on the result. We have sacri- 
ficed conceptual clarity in the pursuit of instructional effectiveness here 
by using the demonstrator routines to demonstrate the correct solution 
whenever the student gives a wrong answer. Finally, the fourth mode 
of instruction is the standard IPI instruction we began with, which is a 
mixture of verbal descriptions, static displays, and a small amount of 
practice with delayed feedback. 

Initially we plan to make these modes available to the school and 
to try them out in different combinations before we formalize and test 
specific sequences. For example, with one possible procedure a student 
would take his completed unit pretest to the computer terminal and sub- 
mit the problems he missed to the demonstration program to see what 
the correct algorithm looked like. Second, he would read a booklet 
describing the correct algorithm. Finally, he would try out the algo- 
rithm using .he practice program and consult the booklet whenever ques- 
tions arose. One of our ultimate goals will be a set of prescriptive pro- 
cedures which will place each student into the mix of instructional modes 
that will optimize his learning. 

Such prescriptions would be the basis of a management system 
that could use both previous student history and specific unit information 
to assign a certain student to a certain type of input mode. For example, 
a student who had shown ability in the past for acquiring algorithms from 
verbal descriptions could be assigned a booklet initially, or a particular 
novel algorithm could be introduced to all students via a demonstration. 
Such a system would in no way preclude student control of the options. 
A student could simply be given a choice among available instructional 
modes, if that were deemed advantageous. 
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A COMPUTER-BASED INDUCTIVE APPROACH 
TO TEACHING ELEMENTARY SCFIOOL MATHEMATICS 



Richard Roman 



This paper describes the MATH FUNCTIONS curriculum package, 
designed to teach both mathematical content and problem solving skills. 
The package is one component of an individualized mathematics system 
operating in an elementary school. All instruction in the curriculum is 
accomplished through a computer program implemented on the schools 
computer resource. Other components of the mathematics system teach 
the same content material, giving students a choice of instructional 
media. The problem solving curriculum package teaches more than 100 
behavioral objectives, or one-fourth of the elementary school mathe- 
matics for grades three through six. 

The ability to successfully solve problems should develop inquiry 
skills. The curriculum package provides a structured envrionment 
which encourages problem solving activities such as stating the problem 
clearly, gathering and organizing data, using feedback, formulating and 
testing hypotheses, knowing you have finished, dividing problems into 
subproblems, and combining subsolutions into complete solucions. Stu- 
dents who experience success at solving problems in this structured 
environment increase their sense of competence and their desire to 
solve more problems. 

From our research to date, some conclusions can be made. Both 
mathematical content and problem solving skills can be taught by struc- 
turing the mathematical content as a sequence of problems in which the 
student induces the organizational rule from examples. Appropriate 
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structuring of the environment in which rules are induced teaches prob- 
lem solving skills. Mathematical content which requires simple rules, 
simple stimulus material and for which the student has mastered the pre- 
requisite skills is taught effectively in conjunction with problem solving 
skills. Many students learn mathematical content in an environment 
which also teaches problem solving skills. 

The mathematics curriculum for which this program is designed 
has been described in a book by Lindvall and Cox (1970). For our pur- 
poses it is enough to note that the school curriculum consists of approxi- 
mately 400 objectives that are grouped into levels v/hich correspond 
roughly to grades in school, and which are further grouped within levels 
into areas such as addition, subtraction, fractions, and money. An indi- 
vidualized booklet and coordinated manipulative materials teach each 
objective to those students who have met the prerequisites, but have not 
demonstrated mastery on the objectives as shown on a pretest for the 
area. Instruction in the booklets is programmed. Generally, the rules 
for operations are stated explicitly and followed by many examples on 
which to practice the rules. 

While we accepted the objectives of the individualized curriculum, 
we designed instruction which differed in style from the booklet design. 
We believed that inducing the rules from carefully structured examples 
would give the students deeper undei standing of the material. Our con- 
text seemed perfect for a project in teaching by guided discovery since 
most of the preconditions for such learning were assured by the way 
instruction is managed at Oakleaf (Gagne, :966). Specifically, we are 
nearly certain that the child has met the prerequisites for the objective 
and is in a state of readiness for learning. 




In the IPI-Functions program, we therefore offer many examples 
of the rules required to perform the objective, but never explicitly state 
the rules for the child. Careful sequencing of small inductive steps 
within each objective allows the student to master the objective by dis- 
covering the regularities embodied in those examples and applying these 
regularities to make predictions during self-testing. 

To be more specific, approximately one -third of elementary 
school mathematics curriculum objectives, both at Oakleaf and in more 
traditional curricula, require the student to master a functional relation- 
ship, rtie stude.. is given an element of a domain set and responds with 
an element of the range set. The functional rule always produces a sin - 
gle range element for each domain element. We designed a program 
which would present material of this type to the student and allow him 
to induce the rule by which the function operated. 

An understanding of the rudiments of the program operation is 
essential for interpreting the results reported. (See Figure 8.) Figure 
8 contains a series of displays as a student would see them on the 
cathode- ray tube screen while he was learning to round off to the near- 
est multiple of 10 or 100, an objective identified as "36-F" in Frame 1 
of Figure 8. Frame 3 is the first instructio.nal display. Notice that it 
tells what the student will learn and gives two samples of the function 
for him to assimilate. It also offers him the choice of trying to input 
an element of the domain, "A PLACE VALUE AND A NUMERAL, " or 
of taking a test or saying done, meaning that he has seen enough of this 
level and wishes to change to the next level of problem. 

Allowing the student to control the number of examples and the 
content of the examples is an important feature of the instructional 
design which allows him to formulate and test his own hypothesis through 
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StXKtont rnponm «ppMr follovnop tht 
fhrMKterbkt (•••). Afttr tfM sixxSent 
hat nwte « rttpon« (•♦• **360** • frame 3). 
tfib rnpomt snd avefytMno between tl and 
fh« (est sample I"100 30 BECOMES 0" • frame 3) 

b tnx^. The new tin* r360 IS NOT A ** . 

frame 4) and difectionf are then otn#rated. 



3 

^ YOU ARE LEARNING TO ROUND OFF 

TOTME NEAREST MULTIPLE OF lOOR 100 

10 $9 BECOMES SO 
100 30 BECOMES 0 

TYPE A PLACE VALUE AND A NUMERAL 
OR -^ESr* OR -OONE". 
— * 360 



v y 



1 



YOU WERE WORKING ON OSJECTlVE 36-F 
LAST TIME. 

00 YOU WAMT THE SAME OSJCCTlVE T00AY7 
— YES 



YOU ARE LEARNING TO ROUND OFF 

TO THE NEAREST MULTIPLE OF 100R 100 

10 S9 BECOMES 60 
100 30 BECOMES 0 
360 IS NOT A PLACE VALUE AND A NUMERAL 

TYPE A PLACE VALUE ANO A NUMERAL 

OR •^£sr• or "done". 

10 30 



THE COMPUTER IS WRITINO A PRESCRIPTION. 
THE COMPUTER IS MAKING UP THE PU?ZLE. 



^ YOU ARE LEARNING TO ROUND OFF ^ 
TO THE NEAREST MULTIPLE OF 10 OR 100 

10 S9 BECOMES 60 
100 30 BECOMES 0 
10 36 BECOMES 40 

TYPE A PLACE VALUE AND A NUMERAL 
01•^EST- OR "DOME". 
10O 120 



Figure 8. Sample CRT displays oi a student interacting 
with the IPI-FUNCTIONS program. 
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YOU AnE LEARNING TO ROUND OFF 

TO THE NEAREST MULTIfLE OF 10 OR 100 

10 69 BECOMES 60 

100 30 BECOMES 0 

to 30 BECOMES 40 

100 120 BECOMES 100 

TYPE A PLACE VALUE ANO A NUMERAL 
OR "TEST " OM -DONE". 
••• DONE 



YOU ARE LEARNING TO ROUNO OFF 

TO THE NEAnEST MULTIPLE OF 10 OR 100 

10 69 BECOMES 60 
ICO 30 BECOMES 0 
10 36 BECOMES 40 
100 120BECOMES 100 
10 48 BECOMES SO 
NOT 40 

TYPE A PLACE VAtUE ANO A NUMERAL 
OR"'TEST ' OR "OONE", 
TEST 



YOU ARE LEARNING TO ROUNO OFF 

TO THE NEAREST MULTIPLE OF 10 OR 100 

10 69 BECOMES 60 

100 30 BECOMES 0 

10 36 BECOMES 40 

100 120 BECOMES 100 

YOU MUST TAKE MORE TESTS BEFORE 
SAYING "OONE" 

TYPE A PLACE VALUE ANO A NUMERAL 
OR "TEST" OR "OONE". 
TEST 
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YOU ARE LEARNING TO ROUND Off 

TO THE NEAREST MULTIPLE OF 10 OR 100 

10 69 BECOMES 60 

100 30 BECOMES C 

10 36 BECOMES 40 

100 130 BECOMES 100 

10 45 BECOMES 60 

TYPE "OONE" OR THE ANSWER 
TO 10 11 BECOMES 
10 



YOU ARE LEARNING TO ROUND OFF 

TO THE NEAREST MULTIPLE OF 10 OR 100 

10 69 BECOMES 60 

100 30 BECOMES 0 

10 3e BECOMES 40 

100 120 BECOMES 100 

TYPE "OONE" OR THE ANSWER 

TO 10 45 BECOMES 

«««40 
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YOU ARE LEARNING TO ROUNO OFF 

TO THE NEAREST MULTIPLE OF 10 OR 100 

10 69 BECOMES 60 
100 30 BECOMES 0 
10 36 BECOMES 40 
100 120 BECOMES 100 
10 45 BECOMES 50 
10 It BECOMES 10 
GREATI 

TYPE "OONE" OR THE ANSWER 
TO 100 880 BECOMES 

poo 



Figure 8 {Cont'd)* Sample CRT displays of a student interacting 
with the IPI-FUNCTIONS program. 
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VOU ARE LEARNING TO ROUNO O^f 

TO THE NEAREST MULTIPLE Of 10 OR 100 

10 M BECOMES to 
100 SOIECOMISO 
10 M BECOMES 40 
100 120 BECOMES 100 
10 4$ BECOMES 60 
10 11 BECOMES 10 

too a80BEco*;ufoo 

OREATI 

T\'f E -TXJNE- OR THE ANSWER 
TO 10 2C6 BECOMES 
•••CONE 



15 

/ X 

f YOU ARE LEARNING TO ROUNO OFF i 
TO THE NEAREST MULTIPLE OF 10 OR 100 

100 MOO BECOMES 0000 
10 B)00 BECOMES 6400 

lyn A PLACE VALUE ANO A NUMERAL 
OR Tf rr OR -OONE-. 
•••STOf 
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TWl -YEr TO SEE A NEW mZZLC. 

TYPE -NO" AT THE CND OF THE fCRIOO. 
• •• YU 
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^ PLEASE WAIT. ^ 
THE COMPUTER IS WRITING A PRCSO^imON. 



14 



PLEASE WAIT. 

TKC CO»J>UTER IS WRITINO A PRESCRIPTION. 
THE CCAVUTER IS MAKING UP THE PUZZLE. 



17 



PLEASE WAIT. 

THE COMPUTER IS WRITING A PRESCRIPTION. 

GOOOBYE NOW. 

COME BACK TOMORROW. 



Figure 8 (Cont'd). Sample CRT displays of a student interacting 
with the IPI-FUNCTIONS porgram. 
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the use of appropriate examples. The student may stay in the example 
phase of the program as long as he likes. 

In Frame 3, the student has misunderstood tixe domain, and 
F rame 4 explains that his input is incorrect. In Frames 4 and 5 he 
succeeds in making correct domain inputs and the computer displays 
the corresponding range elements. The student then tries to go on to 
a new level, which is illegal until he has taken some tests (see Frame 7). 
In Frame 8 a test item is generated randomly from the domain set. The 
student responds incorrectly and gets feedback in Frame 9. He con- 
tinues to test himself in Frames 10, 11, and 12. 

Another aspect of the instruction is illustrated here. The stu- 
dent may take tests as long as he likes. Hopefully^ he will do so until 
he is certain that he understands the rule. The program is adaptive 
here since the number of tests required depends on the information con- 
tained in each item and the particular students' ability to assimilate the 
information. In addition, we feel it is important for students to exercise 
control over their own learning. An important aspect of self-controlled 
learning is the ability to know when one understands, is finished, and 
re<idy to move on. 

In Figure 8 we examined the instructional features that recur 
with each function. In summary these wtre: the samples of the function; 
the consistent format of presentation; feedback on inputs; feedback on 
tests; and the opportunity for the student to control the number of inputs 
and the number of tests in a way that is consistent with the hypotheses 
that he is testing, and his own preferred working style. Students can 
and do make meaningful variations in their use of these features, and 
individuals change their behavior with respect to these options as they 
use the program over time. 
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We have spoken about the instruction to help a student induce a 
single rule. We call that instruction one puzzle* and the full instruction 
for a single objective requires two to 12 puzzles, corresponding approxi- 
mately to single pages in a test* Figure 9 shows all the puzzles used to 
teach a single objective on buniiiiing fractions and comparing the result 
to one* 

A glance at Figure 9 will give an idea of the way the difficulty of 
a function changes between puzzles. In this particular objective the 
student learns to sum two fractions with the same denominator and com- 
pare that sum to one. In the curriculum the comparison is always done 
w'ith tlie fraction on the left and the number 1 on the right* This allows 
the student to respond correctly by simply knowing if the numerator or 
denominator of the sum is larger* He can ignore the 1 in the comparison 
and still pass the curriculum embedded test* Accordingly, puzzle 5 of 
the sequence is equivalent to the curriculum embedded test and mastery 
can be assumed after that puzzle* However, in this and most other objec- 
tives we teach, there are post terminal skills that extend the domain, or 
the generality of the rule learned, beyond the puzzle required for mas- 
tcry. 

When the student begins his work on Objective 42-A, he knows 
how to write fr?vptions equivalent to two wholes (Objective 41-D). He 
can also, given two indicated sums, write greater than, less than, or 
equal to benveet; the sums to complete the number sentence (Objective 
33-F)* He has not learned to add two fractions with the same denomina- 
tor. Instruction in the program begins with puzzle three, with the stu- 
dent comparing a single fraction to one* If he cannot figure out what is 
required, he is given a similar problem at puzzle two, but without the 
fraction sign* If he masters, he is advanced to puzzle four, where he 
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Fu22l« 1. Remedial Lwcl 

Domain: Two nombcrs up to 9 with a "7" in between 
Range: The st^ns > , <, and ■ . 

Rule: Type the signt which mutt replace the "7" \o maKC a Uue 
statenK'nt. 

Description for Student;: 

YOU ARE LGAHNING TO ADD FRACTIONS WITH THE 
SAME DENOMINATOR AND COMPARE TO 1 

USE < FOR LESS THAN 
> FOR MORE THAN 
AND -FOR EQUAL TO 

27 2 BECOMES - 
WG BECOMES < 
8? 4 BECOMES > 



Puzzle 2, Remedial Level 

Domain: Two numbers up to 99 with a "7" in between 

Range: The signt > , < . and • . 

. Rule: Type the iign which must replace ihc "7" to make a Uuc 
statement. 

Oesaiption for Student: 

YOU ARE LEARNING TO ADD FRACTIONS WITH THE 
SAME DENOMINATOR AND COMPARE TO 1 

USE < FOR LESS THAN 
> FOR MORE THAN 
AND FOR EQUAL TO 

36? 8 BECOMES > 
4? 87 BECOMES < 
85 7 85 BECOMES - 



Figure 9. Instructional puzzler for adding fractions 
and comparing the sum to one. 



60 



Pu2z!c 3, Entry Level 

Domain: A fraction with denominator less than 100 and numerator 
less than 100 and less than twice thu denominator. 

Range: The signs " < " ■ " and " > ." 

Rule: Type the sign which must replace the lo make a true 
statement. 

Description for Student: 

YOU ARE LEARNING TO ADD FRACTIONS WITH THE 
SAME DENOMINATOR AND COMPARE TO 1 

USE < FOR LESS THAN 

> FOR MORE THAN 
AND • rOH EQUAL TO 

1/16 BECOMES < 
22/22 BECOMES- 



Puzzle 4, Instructional Level 

Domain: Two fractions as in Puzzle 3, with a "+" in between. 

Range: Fractions with the same denominator up to 100 and 
numerators less than twice the denominator and less 
than 100. 

Rule: The numerator of the range element is computed by 
summing the numerators of the two domain functions. 
The denominator is the same as the denominator of the 
fractions. 

Description for Student: 

YOU ARE LEARNING TO ADD FRACTIONS WITH THE 
SAME DENOMINATOR AND COMPARE TO 1 

FRACTIONS MUST HAVE THE SAME DENOMINATOR 

15/11 + 1/11 BECOMES 16/11 
19/20+ 17/20 BECOMES 36/20 



Figure 9 (Cont'd). Instructional puzzles for adding fractions 
ar>d comparing the sum to one. 
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Puzzle 5. Criterion Level 

Domain: Two fractions as in Puzzle 3, with a ' V in between. 

Range: Fractions as in Puzzle 4 followed by one of the signs 
"< r "-"or">." 

Rule: The fraction is computed as in Puzzle 4, and the sign is 
that which must replace the "/" of the sum in order to 
make a true statement. 

Description for Student: 

YOU ARE LEARNING TO ADD FRACTIONS WITH THE 
SAME DENOMINATOR AND COMPARE TO l' 

USE <-FOR LESS THAN 

> FOR f<K)RE THAN 
AND « FOR EQUAL TO 

FRACTIONS MUST HAVE THE SAME DENOMINATOR 

15/11+ 1/11 BECOMES 16/11 > 
19/20+ 17/20 BECOMES 36/20 > 
2/17+ 5/17 BECOMES 7/17 < 



Puzzle 6. Posl'Cr iter ion Level 

Domain: Fractions as in Puzzle 3, with a "+" in between. 

Range: The signs " < /' " = " and " > ." 

Rule: The fractions are summed as in Puzzle 4. The range 

element is tbe sign needed to replace the "/" of the sum 
in order to make a true sentence. 

Description for Student: 

YOU ARE LEARNING TO ADD FRACTIONS WITH THG 
SAME DENOMINATOR AND COMPARE TO 1 

USE < FOR LESS THAN 

> FOR MORE THAN 
AND = FOR EQUAL TO 

FRACTIONS MUST HAVE THE SAME DENOMINATOR 

18/23 + 21/23 BECOMES > 
1/4 + 2/4 BECOMES < 



Figure 9 (Cont'd). Instructional puzzles for adding fractions 
and comparing the sum to one. 
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discovers how to add two fractions with common denominators, and then 
moves up to the terminal skill, puzzle five. The post-mastery skill 
involves generalization of the techniques. 

Many dimensions of the program are interesting for evaluation 
purposes. We have used the program in the school for one year now, 
and have taught 150 children, a total of 530 different lessons. We col- 
lect data on every display and input the students see, and analyze them 
with the goal of improving the instructional algorithm and the sequencing 
of levels for the various objectives. We are presently undertaking more 
formal evaluation efforts; here we discuss the preliminary impressions 
from the data. 

Results 

We attempted to teach a full spectrum of objectives to third, 
fourth, fifth, and sixth graders. During the year, it has become clear 
that certain kinds of objectives lend themselves to presentation in this 
format more fully than others. Objectives with simple displays to pre- 
sent the domain elements work better than complex ones with many 
components. Thus, reducing a fraction is likely to be easier to teach 
than adding two fractions, simply because the display is less complex. 
Ob'cctives that require much typing for either the range or domain will 
be difficult to teach because the students lack the typing facility required. 
Objectives m which complex calculations are required of the student are 
more difficult because such calculations are subject to error, and be- 
cause such processing slows the rate of instruction. For example, the 
rule about placing decimal points m a three- or four -digit multiplication 
problem will be difficult to teach not because the rule is complex, but 
because the calculation is long and subject to error. 
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One of two outcomes occurs each time a student works on an 
objective in MATH FUNCTIONS: The student masters the criterion puz« 
zle (mastery); or the student does not master the criterion puzzle (non- 
mastery). A further refinement of the outcome occurs for students who 
take the curriculum-embedded test (CET) as a transfer task; The student 
passes the CET (mastery-pass); or the student fails the CET (mastery- 
fail). 

The frequency of the various outcomes is reported in Table 10. 
For the moment, only the overall totals are relevant. Of the total sam- 
ple of 601 outcomes since Spring 1972, 529 (88 percent) were masteries. 
Some students (46) who mastered the objective did not take the CET trans- 
fer test. Of those who did take the test, 38l of 483 (78.9 percent) passed 
it, indicating a high transfer rate from the program-defined criterion 
task to the paper and pencil test. 



TABLE 10 
Outcomes on Objectfves 





N on mastery 


Mastery 


Mastery 
Fail 


Mastery 
Pass 


Overall 


72 (12.0) 


529 (88.0) 


102(21.1) 


381 (78.9) 


Spring 1972 


9 (8^) 


99(91.7) 


23 (25.0) 


69 (75.0) 


Fall 1972 


15(14.7) 


87 (85.3) 


23(33.7) 


59 (66.3) 


VVtntcr & Spring 1973 


42(13.1) 


279 (86.9) 


56(18.4) 


197 (81.1) 


Fall 1973 


6 (8.6) 


64(91.4) 


6 (9.7) 


56 (9a3) 



Note: Numbers in parentheses are pcrcents. 
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Since the MATH FUNCTIONS program changed during the period 
of this evaluation, a partition of the results across time should reveal if 
changes affected the outcomes. Four natural divisions occur: Spring 

1972 with only 20 objectives in a highly supervised setting; Fall 1972 
with 120 objectives in a moderately supervised setting; Winter and 
Spring 1973 with 108 objectives in a lightly supervised setting; and Fall 

1973 with 90 objectives in a lightly supervised setting. Outcomes for 
these periods are reported separately in Table 10. A Chi Square test 

of association of nonmastery versus mastery with time is not significant, 
making the best estimate of the pass rate 88 percent, the overall rate. 
The transfer rate does change over time (x^ = 15. 9, 3 df, p < . 05). 

The only drop in transfer rate occurs between Spring 1972 and 
Fall 1972 when six times the number of teaching sequences were availa- 
ble, and the amount of program supervision decreased. Notice a paral- 
lel drop in the mastery rate at the same time period. Tlie mastery-pass 
rate in Fall 1973 is significantly greater than all other time periods 
{X^=5.6, 1 df , p <. 05), and is best estimated as 90. 3 percent. The 
higher transfer rate results from revision of teaching sequences and 
selective elimination of objectives with high mastery-fail rates. 

Summa ry 

The results reported in the last section demonstrate that mathe- 
matics content can be taught in an environment which encourages prob- 
lem solving. This section summarizes the procedure and th^ conclu- 
sions and points out directions for further work. 

The FUNCTIONS program was designed as a problem solving 
environment. Features were included to teach skills such as stating 
the problem, gathering data, organizing data, using feedback, subdi- 
viding the problem, integrating subsolutions, and knowing when you are 
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finished. Important features include the organized display, the two 
modes (input and test) of interaction, and the lack of compulsion in the 
prog ram. 

Since mathematics contains many examples of functions, much 
of elementary school mathematics can be taught using the FUNCTIONS 
program. Special features of the Individualized Mathematics system 
such as self-paced learning, clear behavioral objectives and prerequi- 
sites, and mastery criteria make it especially adaptable to computerized, 
individual instruction. One hundred objectives of the IndividuaUzed 
Mathematics system were analyzed, and teaching sequences were pre- 
pared for use with the MATH FUNCTIONS program. Each teaching 
sequence consists of an ordered set of puzzles each of which is slightly 
more complex than its predecessor. 

In more than one year of continuous use, students have mastered 
88 percent of the objectives they studied, and have transferred their 
knowledge to a paper and pencil test 78. 9 percent of the time. Improve- 
ments in the program and teaching sequence increased the transfer rate 
to 90. 3 percent during Fall 1973. 

While the overall success of the program in teaching mathematics 
content has been demonstrated, much work remains to be done. Some 
additional lines of investigation include: determining if student aptitudes 
interact with the treatment; developing a model of ideal problem solving 
in the program and demonstrating that students approach the ideal with 
experience; clarifying the methodology for creating teaching sequences; 
and, finally, developing additional programs of a similar nature to 
broaden the scope of application. 
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The success of this one effort at combining the te:*ching of problem 
solving with the teaching of mathematics content suggests that other simi- 
lar efforts should be undertaken. What kinds of programs can be adapted 
to these uses? What other fundamental ideas occurring in mathematics 
can be exploited as functions have been in MATH FUNCTIONS? Addi- 
tional work in this area is addressed toward answering these and other 
important questions. 
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COMPUTER-ASSISTED INSTRUCTIONAL SYSTEM: 
COMPUTER REQUIREMENTS 



Robert J. Fitzhugh 



During the past decade, there has been a continuing and growing 
debate in the computing field between the proponents of extremely large, 
utility-like computer systems each serving many users with diverse 
computing requirements and the proponents of small- and medium-scale 
systems each serving a more limited, homogenous population. The 
issues involved are complex and the final results are not in; however, 
the weight of the argument during the past few years seems to have 
drifted toward the small system concept although some in the computing 
field might disagree. Those who favor the large utility argue that there 
are economies of scale in computing just as there are in electric power 
generation and that a single, very large system is far more cost effec- 
tive than multiple, small systems each with its own support staff and 
facility. In addition, it is argued that a large system can offer a power- 
ful and vc ried range of computing services, some of which simply can- 
not be provided by a smaller system. Large-scale number crunching 
is an example. The small computer enthusiasts counter by pointing to 
the dramatic decline in the cost of small computers that is making them 
increasingly cost effective. Microcomputers the size of a pack of ciga- 
rettes are already available for under $200. The small computer enthu- 
siasts argue that the very large systems that seek to be all things to all 
users have grown cumbersome, overly complex, and difficult to manage. 
A substantial amount of the memory and processing power of these 
machines is often consumed by massive general-purpose operating 
systems. 
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It is clear that this small versus large computer issue will not 
be settled for some time. In fact, a probable outcome is that neither 
will prevail and that there will be a mix of large and small computers, 
depending upon their application. At the Learning Research and Develop- 
ment Center, the application is computer assistance to the elementary- 
school, and a major question is what is the most reasonable means of 
providing this computer assistance? Should it be through a tie-in to a 
large, remote, general-utility system, or can a small computer system 
located on site provide the required services? 

The Learning Research and Dev;:lopment Center hc.s maintained 
a computer research program since 1964 with the early years devoted 
to the development and use of a computer-controlled, on-line psycho- 
logical research laboratory. Through developing and operating the 
On- Line Laboratory, considerable experience was acquired in the use 
of small computers, on their limits and on their use as an educational 
tool that can be made highly adaptive to individual differences. It was 
felt that existing small-computer-based educational systems had not 
fully exploited the powers of the small computer and that c system could 
be developed which could provide a broad range of computer services 
and which could satisfy the computing needs of a single school in a cost- 
effective manner. 

To investigate this and other hypotheses about the roles a com- 
puter might play in a school, a small computer and associated peripherals 
were acquired and were housed in a modified van at a school site* The 
van provides a controlled en\ironment and the option of easily relocating 
the system. The system requires approximately 250 to 300 square feet 
of floor space for a comfortable configuration and five tons of air condi- 
tioning. The current van configuration with office space for the support 
staff is shown in Figure 10* 
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The system was located at the school primarily to avoid the tele- 
phone system and telephone line charges although there is the less tangi- 
ble factor of totaJ local control over the computer resource which is not 
possible with a remote system. Although the line charges that would be 
incurred with the current terminal configuration would be large but not 
prohibitive, the system was located on site to retain the option of up- 
grading to more powerful terminals, possibly with very high data rates. 

The major hardware components of the system were purchased 
from a number of vendors or were designed and constructed by LRDC as 
no single manufacturer at that time offered all of the items that were 
required. It is important to note, however, that no unusual modifications 
were made either to the computer or to the associated hardware so that 
the hardware remains comparable to other similar small- and medium- 
scale systems that are now readily available from computer manufac- 
turers. Figure 11 depicts this hardware configuration. Ttie important 
elements are the DEC PDP-15 computer with 32 thousand words of 
memory, a drum memory unit specially modified to increase the rate at 
which data can be transferred between it and the computer memory, two 
disk units which together provide 50 million characters of storage space, 
and a terminal controller capable of controlling up to 64 terminals of 
varying types and data rates both on site and remote telephone dial-up. 

While the hardware is relatively standard, it is the time-sharing 
software, called ETSS for Experimental Time-Sharing System, that is 
the significant contribution by offering features, services, and interactive 
capabilities presently available only on large-scale computer time-sharing 
systems. Although many of the characteristics of ETSS were implemented 
in response to application-specific requirements, ETSS is a jeneral- 
purpose time-sharing system rather than a special-purpose system 
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designed only for an educational environment. This decision to develop 
a general-purpose system places LRDC at variance with several similar 
projects at other institutions and was based on four primary considera- 
tions: 

1. Hie system was developed primarily as a research venture 
rather than as a prototype of a system that might be replicated and pro- 
duced commercially. As a research tool, the ability to meet changing 
requirements was essential. It was felt that the researcher should 
encounter as few artificial constraints as possible even though the price 
of generality is often increased programming difficulty, 

2. llie system must support several dissimilar applications. 
Although computer -as sis ted testing and instruction are nearly identical 
from a computer point of view, the data management and retrieval appli- 
cation posed an entirely different set of requirements. Rather than 
attempt to define and isolate common requirements that could be incor- 
porated in a "special-purpose" system, the development of a general- 
purpose system was the cleanest solution. 

3. The development of a special-purpose system would most 
probably preclude the use of commonly available and widely understood 
programming languages such as FORTRAN or BASIC. A programming 
language would need to be defined. and implemented, this step was felt to 
be a major one which vould draw heavily on available resources and 
detract from the primary objectives of the project. In addition, there 
was a reluctance to contribute another unusual, one-of-a-kind language 
to the existing Tower of Babel, particularly in the absence of a clearer 
understanding of its potential applications. The development, or at least 
the definition, of an application- specific language will be more appropri- 
ate once substantial in-school experience has been acquired. 
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4. The decision to develop a general-purpose, small computer 
time^sharing system v/as further reinforced by the belief that systems 
much like ETSS will be commercially available from computer manu- 
facturers within several years. A number of systems are currently 
available that offer the language BASIC in a time-shared mode, and 
several multi- language systems are planned or under development. 
The growing success experienced in education markets by small- 
computer companies indicates that future more powerful systems 
such as ETSS will see wide use. 

From a user standpoint, the system behaves very much like other 
general-purpose time-sharing systems now operating on large-scale 
computers although the system of :s a number of interesting special 
features required by this application. A user, who might be a student 
or teacher within the school or a researcher or program developer 
accessing the computer from a remote site, locates an available termi- 
nal and identifies himself to the computer. Once the user has gained 
access, lie is ac a decision point and must specify the services he 
requires or the program he wishes to run. A student within the school 
might enter the name of an instructional or testing program which the . 
system automatically retrieves from disk storage and initiates. A 
teacher or school administrator might wish to interrogate the student 
data base and would request that a data management program be started. 
A computer programmer developing ..dw programs might ask to enter 
the Editor, one of the language processors such as FORTRAN, Focal, 
or T64 or any of a number of other utility or debugging programs. 

From an internal programming standpoint, ETSS offers a wide 
range of programming options and a flexible, yet straightforward, input/ 
output structure. The system masks device idiosyncrasies and offers 
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complete device independence. User programs interface with a virtual 
input/output structure in which "files" are referenced which maybe 
assigned to any appropriate device or dataset. 

For this educational application, the file structure and the ';7ays 
in which data and progran.i may be stored and accessed are very impor- 
tant and several feuttvej^ were provided which are not commonly availa- 
ble on smaller, commercial systems. Programs and data maybe stored 
in datasets which are "cataloged" in a directory unique to each user, or 
they may be stored in a common directory called the "Library, " A user 
can access any program in the Library and may access a dataset in a 
user's private directory if that user has declared that this is permissible. 
In addition, any number of programs and users may simultaneously access 
common disk files without interference. For example, multiple users 
can interrogate and update a common student data base simultaneously 
and without interference, each unaware of the other's actions. Programs 
can test to determine whether or not a specified dataset ts currently being 
accessed, and multiple programs can write to common "log" files which 
can later be sorted and analyzed. Facilities are also provided for the 
accurate timing of input/output events permitting response latency meas- 
urements to be made. Finally, the system has a number of internal, 
self-monitoring features which enable the system designers to collect 
and analyze data on system performance and use. 

User prograns running under ETSS may range .ip to l6K words 
or, roughly, 32K bytes in size. To put this limit in more operational 
terms, the largest instructional program written in FORTRAN that is 
currently operational is more tlian 1,500 FORTRAN statements in length 
with 4,000 additional words of dimensional arrays. The typical instruc- 
tional program falls in the 900 to 1,300 statement range. This 16K limit 




on program size has proved to be reasonable and one that has not inhibited 
or unduly constrained program development. 

The disk requirement is more difficult to determine and varies 
greatly from program to program and from subject matter to subject 
matter. Programs in the language arts area in which textual material 
is displayed tend to have far greater on-line storage requirements than 
instructional programs in mathematics. Beyond citing examples, it is 
difficult to be more precise. The current range is from the 2. 5 million 
characters required by a reading comprehension program to the nominal 
storage requirements of several mathematics programs that generate 
items for display to the student. Student history files can vary in size 
depending upon the position in the school year and the extent to which 
students have had computer exposure. For the Individually Prescribed 
Instruction environment, 4,000 characters per student have proved to be 
a good estimate for planning purposes. From a total project point of 
view, the 50 million characters of on-line storage the system now pro- 
vides appear to be ample for mo3t conceivable applications in a 300- 
student elementary school. 

The experience of the coming year will provide the information 
required to refine these estimates and further define the computer require- 
ments of the elementary school. Ho'.vevcr, with the experience base of 
nearly two years of successful operation in the school, it now seems clear 
that a small, local system is potentially a viable and cost-effective alter- 
native to the large general utility. 
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